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Qualitative Analysis on Stage: Making the Research Process More Public 

Introduction 

The worth of any research endeavor is assessed by a variety of audiences — peers, 
editorial review boards, publishers, grant reviewers, and dissertation committees. Early in 
the introduction of naturalistic inquiry in the field of education, critique of qualitative 
research seemed to come primarily from those quantitatively oriented. Too frequently, 
qualitative research was evaluated against the positivist criteria of validity and reliability 
and was found to be lacking or “soft.” 

In defense of this criticism, many educational researchers, in their eagerness to 
embrace qualitative methods, did not provide adequate and clear justifications for their 
methods, findings or conclusions (Howe & Eisenhart, 1990). As was foreshadowed by 
Guba (1981), “. . .the naturalistic approach is likely to be tarred with the brush of ‘sloppy 
research’. . .” (p.90). Indeed, providing access to the decisions that are made in the 
process of conducting qualitative research is part of responding to the question of 
whether or not the findings are sufficiently credible and trustworthy (Borman, 1985). 

Recently, the source of this critique has changed. “Unprecedented criticism of 
ethnographic or qualitative methods, substance, style, practice, and relevance has 
emerged. The criticism emerges not from the traditional enemies, the positivists who fault 
qualitative research for its failure to meet some or all of the usual positivistic criteria of 
truth, but from the insiders to the ethnographic movement” (Altheide & Johnson, 1994, 
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p.485). 
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Criticism from both inside and outside sources, as well as the proliferation of 
qualitative methods in educational research, has led to considerable controversy about 
standards for the design and conduct of research (Howe & Eisenhart, 1990, p. 2). The fact 
that justifications are often inadequate or unclear is due in no small measure to confusion 
about how to best think about standards for qualitative research design and analysis. 

It is argued here that discussions regarding standards for qualitative research (see 
Goetz & LeCompte, 1984, Ethnography and Qualitative Design in Educational Research ; 
Smith & Glass, 1987, Research and Evaluation in Education and the Social Sciences ; 
Denzin, 1989, The Research Act: A Theoretical Introduction to Sociological Methods) 
have failed to address one very important dilemma — questions concerning the credibility 
and status of qualitative inquiry as related to the privatization of this type of analysis. Put 
another way, criticism stems from our inability to deal with the “art of the science” (see 
Denzin, 1994; Fontana & Frey, 1994). We operate from the basic premise that how 
researchers account for and disclose their approach to all aspects of the research process 
is key to evaluating the work substantively and methodologically. 

Our purpose in this paper is to address some of the strategies that we have 
employed in our work with doctoral students and to offer suggestions for assessing the 
methodological rigor and analytical defensibility of this paradigm. As used in this paper, 
rigor is defined as the attempt to make “data and explanatory schemes as public and 
replicable as possible” (Denzin, 1978, p.7). We offer these strategies as suggestions for 
judging the rigor of products of naturalistic inquiry and as a criterion to be considered by 
the qualitative research community. 
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We are motivated in this endeavor by three observations. First, what exactly does 
it mean when a researcher writes, “themes emerged?” As an example, recently the first 
author of this paper reviewed a manuscript for a major educational journal. In the 
methodology section the author of that manuscript noted, “In analyzing the data and 
identifying patterns, I developed themes, episodes, and subcategories.. . I arrived at the 
subcategories by analyzing my data further.” The reader is expected to take the word of 
the researcher that he/she did a credible job in data analysis — that the themes that 
emerged actually have some congruence (verisimilitude) with the reality of the 
phenomenon studied. 

Secondly, while triangulation, member checks, and other qualitative techniques 
are mentioned frequently in design or methods sections of research articles, rarely is there 
evidence of exactly how these were achieved. They are invoked as if magical incantations 
and the reader must simply believe and trust the researcher — a leap of faith that is 
sometimes hard to accomplish. Referring to triangulation techniques, the author of the 
manuscript noted above wrote, “Additionally, I used triangulation to increase the 
reliability of my research findings. . ..Triangulation of data in this research occurred 
between [sic] the teacher, researcher, student participants and the data sources.” 
Unfortunately, no evidence of this was found in the data analysis presented in this 
manuscript. Verbatim quotes from students were presented, but there was no evidence of 
direct quotes from teachers or “the data sources.” 

Thirdly, rarely are we privy to the interview protocol that is used to collect data. 
Using the same manuscript mentioned above, the author did not provide the reader with 
the interview questions or any hint of those questions. The analysis presented actually 
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leads the reader to wonder what the interview protocols looked like. These observations 
have led us to conclude that in all the discussions of validity in qualitative research there 
is one major element missing — the public disclosure of processes. 

The particular stages of the research process that are the focus of this paper are 
under the umbrella of research design and analysis. We focus specifically on the 
relationship between research questions and data sources, the processes of theme 
development, and triangulation. Examples from dissertations (Brown, 1999; Mickey, 
2000; Roney, 2000) are presented for the purpose of illustrating the utility of this 
approach. 

In our attempt to do this, we follow the lead of Constas (1992) who wrote, “Since 
we are committed to opening the private lives of participants to the public, it is ironic that 
our methods of data collection and analysis often remain private and unavailable for 
public inspection (p.254). Other scholars have said this in different ways. Guba (1981) 
wrote, . .while practitioners of naturalistic approaches have been reasonably 
introspective about what they do, they have not made systematic efforts to codify the 
safeguards that they intuitively build into their inquiries” (p.76). Agreeing with this 
sentiment, Oakley (1981), in discussing the interviewing process, commented, . .behind 
every closed front door there is a world of secrets” (p. 41). 

Indeed, a key part of qualitative research is how we account for ourselves; how 
we reveal that “world of secrets.” Good naturalistic inquiry shows the hand and opens the 
mind of the investigator to his/her reader. “The effort may not always be successful, but 
there should be clear ‘tracks’ indicating the attempt has been made” (Altheide & Johnson, 
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Validity in Qualitative Research: A Review of the Debate; What is Missing 

The literature is replete with discussions of standards for assessing the quality and 
rigor of qualitative research. Early proposals addressing concerns for validity in 
ethnographic or qualitative research focused on foundational metaphysics. They were 
grounded in the issues and concerns to which conventional inquiry typically addressed 
itself. Early proposals for validity criteria focused on four issues — internal validity, 
external validity, reliability, and objectivity — that are traditionally addressed in 
quantitative studies. Realizing, however, that those concerns simply could not be 
addressed well in naturalistic research, different researchers and theoreticians recast the 
four criteria to meet varying needs that were thought to exist (Lincoln, 2001). 

Among the first scholars to address this issue were Guba and Lincoln (1981 ; 

1982) and Lincoln and Guba (1985). Retreating from the language of experimental 
inquiry, Lincoln and Guba restated the rigor criteria as trustworthiness criteria (see Table 
1). Reflecting back, Lincoln (2001) noted, “At the time we devised the trustworthiness 
criteria (Guba & Lincoln, 1982), we realized they were rooted in the concerns of 
positivist inquiry, but were not certain how to proceed with breaking free of those 
mandates” (p.34). These trustworthiness criteria are “well employed in dissertation 
research. . . helping students to understand that fieldwork is a prolonged process, with 
many steps which can be utilized to ensure that unexplored bias does not creep into the 
work, and that sufficient credibility and plausibility checks are carried out to ensure that 
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Table 1: Quantitative and Qualitative Criteria for Assessing Research Quality and Rigor 



QUANTITATIVE TERM 


QUALITATIVE TERM 


STRATEGIES EMPLOYED 


Internal Validity 


Credibility 


• 


Prolonged engagement in 








field 






• 


Use of peer debriefing 






• 


Tri angulation 






• 


Member checks 






• 


Time sampling 


External Validity 


Transferability 


• 


Provide thick description 






• 


Purposive sampling 


Reliability 


Dependability 


• 


Create an audit trail 






• 


Code-recode strategy 






• 


Tri angulation 






• 


Peer examination 


Objectivity 


Confirmability 


• 


Triangulation 






• 


Practice reflexivity 



the case matches the constructions of individuals and groups in the context” (Lincoln, 
2001, pp. 34-35). 

Other constructions of validity for qualitative inquiry do not resemble the more 
traditional positivist typology offered by Guba and Lincoln (1982) — they are alternatives 
to the more conventional/traditional approach offered by positivism (see Lincoln, 2001 
for a more complete discussion of these validities). Five general standards for validity 




8 



were discussed by Howe and Eisenhart (1992) in the first major handbook of qualitative 
research, The Handbook of Qualitative Research in Education , edited by LeCompte, 
Millroy, and Preissle (1992). Warning that any general standards for evaluating 
educational research would have to be very abstract, Howe and Eisenhart (1992) 
proposed the following five criteria: (1) ensuring a fit between research questions, data 
collection procedures, and analytic techniques; (2) ensuring the effective application of 
specific data collection and analytic techniques; (3) being alert to and cognizant of prior 
knowledge; (4) being cognizant of both internal and external value constraints; and, (5) 
assessing a study’s comprehensiveness. 

Silverman (1989) offered the research community six rules for qualitative 
research: (1) don’t mistake critique for a reasoned alternative, (2) avoid treating the 
actor’s point of view as an explanation, (3) recognize that the phenomenon always 
escapes, (4) avoid choosing between all polar oppositions, (5) never appeal to a single 
element as an explanation, and (6) understand the cultural forms through which “truths” 
are accomplished. In a similar vein, Athens (1984) offered three scientific criteria for 
evaluating qualitative studies: theoretical import, empirical grounding, and scientific 
credibility (see pp. 26 1-266 for discussion of these criteria). 

Whether we remain faithful to the concerns of positivist metaphysics or depart 
from these validity and reliability criteria we cannot ignore the caution offered by Howe 
and Eisenhart (1990) that the common strategy of grounding qualitative research in the 
positivist paradigm creates a “procrustean bed” for itself by assuming it must coexist with 
positivism and it must define itself as positivism’s opposite. There is no doubt that the 
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traditional criteria for methodological adequacy were formulated and essentially owned 
by positivism (Altheide & Johnson, 1994). 

Validity issues in qualitative research will not go away. “Validity is virtually 
synonymous with trouble these days. It is trouble for the simple reason that, although 
originally conceived as a requirement for rigor in the pursuit of conventional inquiry, it 
has been carried over into phenomenological, naturalistic, participative, poststructural, 
postmodern, and other case study forms of inquiry” (Lincoln, 2001, p.25). While the 
question has been answered with a set of procedures that satisfy both positivist and 
experimental researchers, it is far from settled in qualitative research. 

Interestingly enough, all of these “varieties of validities” (Lincoln, 2001) have 
failed to address the issue of publicly disclosing decisions made during the research 
process — demonstrating the methods and processes “by which raw data were collected 
and the processes by which they were compressed and rearranged so as to be credible” 
(Lincoln, 2001, p.25). We offer this as a new criterion for consideration by the qualitative 
research community. It is our belief that public disclosure will afford us the ability to deal 
with the “science of the art” of qualitative research. 

The Documentational Tables 

As note earlier, the problem is that qualitative researchers do not always provide 
their readers with detailed explanations of how research questions are related to data 
sources, how themes or categories are developed, and how triangulation is accomplished. 
While researchers claim to utilize triangulation, member checks, and discuss the 
development of the themes presented, what is actually done is often anyone’s guess. Most 
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studies do not reveal these inner-workings and good writing can cover up awkwardly 
collected and poorly documented fieldwork. 

Three dissertation studies (Brown, 1999; Mickey, 2000; Roney 2000) are used to 
illustrate the utility of the tables presented herein. Conducted in middle schools, the first 
focused on teaming and advisory programs in middle schools, the second looked at 
characteristics of effective middle school teachers, and the third study investigated the 
principal as change agent and instructional leader. 

Two disclaimers should be kept in mind when considering application of the 
approach described here. First, no claim is made that this approach assures validity or 
trustworthiness. Secondly, the primary value of this approach rests on its potential to 
encourage researchers to make analytical events open to public inspection. 

Data Collection: Designing Interview Questions that Address Research Questions 

Research design deals with answering who, what, when, where, how, and why 
questions. Yin (1994) discussed design as “the logic that links the data to be collected 
(and the conclusions to be drawn) to the initial questions of the study” (p.18). Keeping in 
mind that research questions provide the scaffolding for the investigation and the 
cornerstone for the analysis of the data, the process of forming interview questions on the 
basis of what truly needs to be known is a fundamental step. In-depth interviewing as a 
method of gathering information is a way to correlate etic issues (see Stake, 1 995) with 
emic issues (see Hamel, 1993). The following matrix, Table 2, presents the reader with an 
example of three major research questions and two subquestions which served as the 
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Table 2: Research Questions in Relation to Interview Questions 



Research Questions 


Interview Questions 


1) What are the characteristics identified 
by principals, teachers, and students 
that middle level teachers need to 
possess in order to be effective in 
teaching young adolescents? 


P2, P3, P4, P6 
T2, T5, T6, T7, T8, T9, T10 
SI, S4, S5, S6, S7, S8, S10, SI 1 


a) Are there common identifications 
between and among the three 
groups of participants (middle 
school principals, teachers, and 
students) with regard to 
characteristics middle level teachers 
need to possess in order to be 
effective in teaching young 
adolescents? 


P2, P3, P4, P6 
T2, T5,T6, T7, T8, T9, T10 
SI, S4, S5, S6, S7, S8, S10, SI 1 


b) Are there different identifications 
between and among the three 
groups of participants (middle 
school principals, teachers, and 
students) with regard to 
characteristics middle level teachers 
need to possess in order to be 
effective in teaching young 
adolescents? 


P2, P3, P4, P6 
T2, T5,T6, T7, T8, T9, T10 
SI, S4, S5, S6, S7, S8, S10, SI 1 


2) How do teacher preparation programs 
help or hinder the development of 
middle level teachers and their feelings 
of effectiveness? 


P2, P4, P5, P6, P7 
T2, T3, T4, T5, T6 
S4, S5, S7, S8, S9 


3) How do on-the-job experiences help or 
hinder the development of middle level 
teachers and their feelings of 
effectiveness? 


PI, P3, P4, P6, P7, P8, P9, P10 
Tl, T2, T5, T6, T7, T8, T9, T10 
S2, S3, S4, S5, S7, S8, S10 
Legend: P = principal 
T = teacher 
S = student 
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foundation on which the subsequent interview questions (see Appendix A) were 
designed. To the right of each research question are codes (i.e., P2, T5, S4) referring to 
specific interview questions. P2, for example, indicates the second question from the 
interview protocol developed for the school principals. Constantly revisiting the central 
questions that the researcher hopes to answer is helpful in establishing a base of reference 
for the exploratory interview questions. 

This multisite qualitative case study (Roney, 2000) was devoted to defining 
“effective” as it relates to characteristics of middle level teachers. Semi-structured 
interviews were the primary data gathering source used to help construct the participants’ 
perspectives regarding the research questions (i.e., What are the characteristics that 
middle level teachers need to possess in order to be effective in teaching young 
adolescents?). Because of this fact, it was imperative that the interview questions be 
carefully cross-referenced to the study’s research questions. The researcher (Roney, 

2000) could not afford to conduct interviews and prepare transcriptions only to discover 
that the right questions were not asked. It needs to be pointed out that this type of matrix 
could just as easily show the relationship of other data sources (documents, observations, 
surveys) to the study’s research questions. From our experience with dissertation 
advising, though, the utility of utilizing this matrix has helped to insure that the right 
questions are asked; at least questions that will help answer the study’s main question(s). 

Data Management: Conducting Data Analysis through Code Mapping 

The purpose of analysis is to bring meaning, structure, and order to data. 
Interpretation requires acute awareness of the data, concentration, and openness to subtle 
undercurrents of social life (Marshall & Rossman, 1995). Confronted with a mountain of 
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impressions, documents, transcribed interviews, and field notes, the qualitative researcher 
faces the difficult task of making sense of what has been learned. Denzin and Lincoln 
(1994) called this task the “art and politics of interpretation” (p.500). Van Maanen 
(1988) also noted that the researcher must translate what has been learned into a body of 
textual work that communicates these understandings to the reader. He referred to this 
process as telling the “tales of the field.” The purpose of this process is to present the 
reader with the stories identified throughout the analytical process, the salient themes, 
recurring language, and patterns of belief linking people and settings together. Table 3 
outlines six themes that emerged from the data analysis of a research study (Brown, 

1999) investigating how advisory programs either help or hinder the creation of a sense 
of care and community in middle schools. 

The process of data analysis is eclectic; there is no “right way” (Tesch, 1990). 
Creswell (1994) noted that “data analysis requires that the researcher be comfortable with 
developing categories and making comparisons and contrasts. It also requires that the 
researcher be open to possibilities and see contrary or alternative explanations for 
findings” (p. 1 53). According to Patton (1980), the data generated by qualitative methods 
are “voluminous” and this process of sitting down and making sense out of pages of 
interviews and whole files of field notes can be “overwhelming” (p.297). The purpose of 
Table 3 is to present the reader with the larger, consolidated picture that emerged from 
the “reduction and interpretation of the data collected” (Marshall & Rossman, 1989, 
p.l 14). As the reams of data were brought into manageable chunks (see Table 3, First 
Iteration) and meaning and insights were brought to the words and acts of the participants 
involved, several themes (see Table 3, Second Iteration) were generated. Tesch (1990) 





14 



Table 3: Code Mapping: Three Iterations of Analysis 

CODE MAPPING FOR ADVISORY PROGRAMS 

(Research Questions 1, 2 and 3) 



RQ#1 : A Sense of Community 
And Care? 



RQ#2: Structural/Procedural 
Components and the 
Sense of Community? 



RQ#3: The Effect of Advisory 
Programs on Teachers 
and Students? 



(THIRD ITERATION: APPLICATION TO DATA SET) 



Creating Community in a State of Bureaucracy: 
The Paradox of Producing and The Process of Praxis 



(SECOND ITERATION: PATTERN VARIABLES) 



1 A. Caring is Woman’s Work 



IB. Fear of the Affective 

Domain: For Some 
Yes and For Some No 



2A. Battle Lines: Administrative 
Support Vs. Teacher 
Resistance 

2B. Student Mingling or 

Teacher Meddling? 



3 A. From Attention Provider 
to Detention Giver 



3B. No Matter What Happens, 

Something Good Seems 
to Result! 



(FIRST ITERATION: INITIAL CODES/SURFACE CONTENT ANALYSIS) 



1 A. Gender Issues/Equity? 

1 A. Nurturing Ability? 

IA. Male Advisors? 

1 A. Only Women Can Care? 

IB. Affective Vs. Cognitive 
IB. Waste of Instructional Time 
IB. Uncomfortable/Touchy-Feely 
IB. Interpersonal/Family-like 



2A. Organizational Structure 
2A. Accountability ?/Training? 
2A. Support Vs. Dissatisfaction 
2A. Scheduling/Resources? 

2B. Student Interest/Sharing? 
2B. Trust/Respect 
2B. Unreceptive/Bewildered 
2B. Supportive Relationships 



3A. Discipline Problems 
3 A. Demanding/Disruptive 
3 A. Positive and Proactive 
3A. Focused and Patient 

3B. Connections/Interactions 
3B. Feel Better/Get Help 
3B. Know Personally/Easier 
3B. Communication/Difference 





DATA 



DATA 



DATA 



DATA 
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called this process “de-contextualization” and “re-contextualization.” The Third Iteration 
(see Table 3) brings the analysis to a level of hypothesis or theory development. 

Table 3 attempts to examine interpersonal support structures through the lens of advisory 
programs. Underlying patterns that form theoretical constructs about how relationships 
can be fostered and developed in middle school advisories and how these programs can 
promote a caring, community atmosphere were investigated. The research questions that 
were answered included: (1) How do advisory programs help or hinder the creation of a 
sense of community and care for students and teachers?, (2) How do the structural/ 
procedural components of an advisory program hinder or enhance the creation of a sense 
of community?, and (3) What do teachers and students say is the most significant effect 
of advisory programs on school? Through the voices of the interviewees Table 3 
highlights the preconceived notions, fears and findings of actual advisories, the levels of 
support and resistance associated with such programs, and the pros and cons involved 
when establishing interpersonal advisor-advisee relationships. 

Merriam (1988) and Creswell (1994) recommended simultaneous data collection 
and analysis for generating categories and building theories. As data were being coded 
(first iteration), the responses were compared within categories and between categories 
(second iteration). This technique, described in detail in the work of Glaser and Strauss 
(1967), is referred to as constant comparative analysis. Constant comparative analysis 
occurs as the data are compared and categories and their properties emerge or are 
integrated together. Utilized in this study, this process led to the generation of the 
theoretical properties of the categories and was intended to generate the findings of the 
study. 
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Constant comparative analysis aided in identifying patterns, coding data, and 
categorizing findings. Miles and Huberman (1994) suggested that initial coding be 
conducted to find conditions among the participants, as a method of pointing to 
regularities in the setting (first iteration). As Bogdan and Biklin (1982) explained, “ . . . 
certain words, phrases, patterns of behavior, subject’s ways of thinking, and events repeat 
and stand out” (p. 166). In this study, the words and phrases generated from the 
formulated patterns served as the coding categories. This process of identifying and 
“tagging” data for later retrieval and more intensive analysis is called “code mapping” 
(Seidel, Kjoiseth, & Seymour, 1988). Patton (1990) stated that the first part of content 
analysis is to examine what is there and label it. Indeed, Goetz and LeCompte (1984) 
agreed that the goal of qualitative research is “to reconstruct the specific categories that 
participants used to conceptualize their own world view” (p.6). The designation of these 
categories provided the investigator with a manageable way of describing the empirical 
complexities of summarizing hundreds of pages of interview transcriptions (Constas, 
1992). 

The qualitative researcher was accountable for indexing the code map in Table 3 
and also recording the steps involved in Glaser and Strauss’s (1967) “constant 
comparative method.” By making all aspects of the analysis process open to public 
inspection, it was hoped that the chain of evidence created and the “audit trail” 
constructed (see Lincoln & Guba, 1 985) would strengthen the dependability and 
reliability of this research. Given that the goal of qualitative research is “to reconstruct 
the specific categories that participants used to conceptualize their own world view” 
(Goetz & LeCompte, 1984, p.6), the researcher is responsible to document the procedures 
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used to generate categories. Through the use of constant comparative analysis to identify 
the ethic of care and the creation of a sense of community, the goal of generating 
plausible categories, properties, and characteristics of advisory programs was achieved. 
The goal was not to develop “grounded theory” (see Woods, 1985), but to present a 
viable interpretation of the findings collected. 

Findings and Data Triangulation: Methods of Verification 

A common criticism directed at qualitative research is that it fails to adhere to 
canons of reliability and validity (LeCompte & Goetz, 1982). Internal validity is 
concerned with how trustworthy the conclusions are that are drawn from the data and the 
match of these conclusions with reality, while external validity refers to how well 
conclusions can be generalized to a larger population. Ratcliffe (1983) states, “data do 
not speak for themselves; there is always an interpreter, or translator” (p.149). 

The aim of the research offered in Table 4 was to investigate the principal as 
change agent and instructional leader. Shown are the complexities of variables and 
interactions that are so embedded in the data derived from the setting that it cannot help 
but be valid (Marshall & Rossman, 1989). Multiple sources of evidence, an established 
chain of evidence, pattern-matching, replication logic in multiple-case studies, use of 
proper case study protocol, a developed case study data base, and member checks all add 
to the validity and reliability of this study. Table 4 shows how multiple sources of data 
collection as well as multiple voices — the voices of teachers, teacher leaders, and 
administrators — were used to triangulate the data for this study. Sources of data 
collection consisted of individual face-to-face interviews with key informants, 
observations, discussions that resulted from informal visits to classrooms, a questionnaire 
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Table 4: Matrix of Findings and Sources for Data Triangulation 



Sources of Data: 


I = Interview 
Q = Questionnaire 


O = Observation 
D = Documents 


Major Findings 




Sources of Data 




I 


o 


Q D 


Category I: Instructional Leadership 
1 . The principal's strong instructional leadership had 


X 


X 


X X 


a significant influence on the success of pedagogical 
restructuring. 

2. The principal provided the necessary resources to 


X 


X 


X 


support change. 

3 . Extensive professional development was a key factor 


X 


X 


X X 


for successful pedagogical restructuring. 

The development of teacher leadership facilitated 




X 


X 


pedagogical restructuring. 









Category 2: Accountability 



The principal held teachers more accountable for student 
learning than any other group. 


X 


X 


Teachers resented having the greatest share of accountability 
for student learning. 

Teachers held the principal accountable to maintain a 


X 


X 


disciplined school climate and were critical of the 
principal's refusal to be responsible for how students 
behaved. 


X 


X 


Prior to restructuring, teachers were held accountable for 
good classroom management— not student learning. 


X 


X 



Category 3: Collegiality 

8. Teacher leaders involved veteran teachers in the decisions X X 

that affected them and reduced their resistance to change. 

9. Team building and development of leadership in teachers X X 

promoted cooperative relationships among teachers. 



Category 4: The Milieu: The School Environment 



10. Teachers and the principal were polarized because of two very X 

different philosophies: Good teaching prevents discipline problems 
versus teachers need discipline to accomplish good teaching. 


X 




1 1 . The lack of a disciplined school environment was the major barrier X 
in the restructuring process. 


X 


X 


12. A more student-centered environment was a direct outcome of X 

pedagogical restructuring. 


X 


X 



Category 5: Change: An Evolutionary Process 



14. New teachers embraced change; veteran teachers resisted change. XXX 

15. The principal was the catalyst for change because change was X X 

mandated and teachers could not opt out of the process. 
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that was administered to a select group of teachers, and examination of a wide assortment 
of documents. Table 4 shows the major findings of this study listed under five categories 
and the four sources of data collection. Each data source provides corroborative evidence 
to verify information obtained by other methods. Each finding listed in Table 4 is 
corroborated by at least one other source of data, and in several cases, three or more 
sources of data. In this particular study the use of multiple sources of data collection as a 
form of triangulation prevented reliance exclusively on a single data collection method 
and thus neutralized any bias inherent in a particular data source. 

In this study, triangulation of the interviews with questionnaires, observations, 
and document analysis, as well as triangulation of the interviews with one another 
(teachers<-»teacher leaders<-»administrators), rendered a holistic understanding of the 
situation and generally converging conclusions. As Fielding and Fielding (1986) stated, 
“Triangulation puts the researcher in a frame of mind to regard his or her own material 
critically, to test it, to identify its weaknesses, to identify where to test further doing 
something different” (p.24). 



Concluding Discussion 

After decades of academic and paradigmatic politics, qualitative research finds 
itself in an astonishing position. This is unanticipated by all, especially by those closest to 
it, who were for so many decades accustomed to its devalued, unappreciated, marginal 
status (Altheide & Johnson, 1994). We noted at the beginning of this paper that the 
heightened use of qualitative methods has spurred interest in developing formal standards 
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for assessing the validity of qualitative research. Such standards are important because of 
the legitimacy they afford the research approach. But we contend that these standards 
have missed the mark by not including public disclosure of methods as a criterion. 

The primary point we argued for is the accountability of the researcher in 
documenting the actions associated with establishing internal validity (triangulation), 
theme development, and the relationship between research questions and data sources. 
Generally speaking, this article is concerned with issues related to the integrity of 
qualitative research. The purpose of these tables is to enhance the opportunity for 
criticism and public inspection of qualitative studies — to encourage analytical openness. 

In calling for the public documentation of category development, Constas (1992) 
wrote, “If qualitative research is to gain the acceptance of a broad audience, and not only 
those inclined to accept qualitative inquiry as valid, individuals engaged in qualitative 
empirical research must begin to make all phases of their investigations open to public 
inspection. Extensive methodological and analytical information must be provided if a 
community of researchers is to perform the desired critique and assessment of a given 
research project. The absence of the opportunity for public inspection will likely result in 
suspicion, naive acceptance, or outright dismissal among a community of readers, none 
of which is desirable or necessarily warranted” (p.266). Peshkin (2000), discussing 
interpretation in qualitative research, wrote, “To be forthcoming and honest about how 
we work as researchers is to develop a reflective awareness that, I believe, contributes to 
enhancing the quality of our interpretive acts” (p.9). 

We share the danger of reducing the practice of qualitative research to technical 
issues to be resolved by cookbook methods. Indeed, to talk about tabular strategies, such 
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as those in this article, invites charges of simplification and overgeneralization. This is 
not our intention. However qualitative researchers address validity in their research, we 
argue that the processes employed in the research must be made more public. The 
qualitative ethic calls for researchers to substantiate their interpretations and findings 
with a public accounting of themselves and the processes of their research. 
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Appendix A 

Interview Questions from Roney (2000) 

INTERVIEW QUESTIONS: PRINCIPALS 



Name: 

Ethnicity: 

Years in Middle Level Education: 
Degree: 



School: Date: 

Gender: Age: 

Years in Administration: 

Concentration: Certification: 



1 . Tell me what it is like to be a middle school principal today? When you selected administration as a 
career goal, did you intend to become a middle grades principal? 

2. In your opinion what makes an effective teacher? How does a teacher get to be effective? How does 
one’s background contribute to one’s effectiveness? 

3. How would you describe an effective middle school teacher? Are there characteristics that middle 
school teachers need to possess or to develop in order to be effective middle school teachers? Is there a 
priority order to this list? 

4. What qualities or characteristics are lacking in the teachers that apply for teaching positions? What 
qualities or characteristics would you like to see in the teachers that you interview? 

5. In your opinion do teacher preparation programs (in their methodology courses, content areas, and 
practica student teaching) help or hinder the development of an effective middle school teacher? 

a. Does the academic/content course work adequately prepare teachers for the classes and the 
subject area/s that they teach here? 

b. Do these programs adequately prepare teachers to deal with students in the affective domain? 

6. With reference to the “earmarks” of developmentally responsive middle schools, what should teacher 
preparation programs be focusing on? 

a. What are the important ideas, principles, or understandings that an effective middle level 
teacher needs to know about? 
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b. What do they need to know about interdisciplinary-thematic organization of 

curriculum? 

c. What do they to know about teaming? 

d. What do they need to know about flexible block scheduling? 

e. What do they need to know about exploratory programs? 

f. What do they need to know about advisory programs? 

g. What do they need to know about transition programs? 

7. What is your opinion of a specialized middle level teacher preparation program?. . . certification?. . . 
licensure? Since Pennsylvania does not have a middle school certification, which do you prefer for 
your middle school, a teacher with secondary certification or one with elementary certification? Why? 

8. Do on-the-job experiences help or hinder a teacher in becoming an effective middle level teacher? 
How? Of the many on-the-job experiences they have in the course of a school year, which do you 
consider to be the most helpful? Please comment. 

9. In order to help teachers become effective, what should professional development programs be 
focusing on? In what types of professional staff development programs have your teachers 
participated? Have any been specifically geared toward middle level education? What were they like? 

10. What do you have here with regard to teacher supervision?... evaluation? ... teacher mentoring? How 
do they work here? Do they help or hinder the improvement of a teacher’s effectiveness? Please 
comment. 

1 1 . What are the signs that a teacher is effective? 

INTERVIEW QUESTIONS: TEACHERS 



Name: 


School: 


Date: 


Ethnicity: 


Gender: Age: 




Years Teaching: 


Years In Middle Level Education: 




Grade/s and Subjects teaching: 






Degree: 


Concentration: 


Certification: 





1 . Tell me what it is like to be a middle school teacher today? When you selected teaching as a career 
goal, did you intend to become a middle grades teacher? 

2. What makes a teacher effective? How would you describe an effective middle school teacher? How 
does a teacher get to be effective? How does one’s background contribute to one’s effectiveness? 

3 . Thinking back to the teacher preparation program at your college/university (in its methodology 
courses, content areas, and practica student teaching), did your program help or hinder your 
development as a middle school teacher? 

a. Did the academic/content course work adequately prepare you for your classes 

and the subject area/s that you teach here? 

b. Did your program adequately prepare you for the adolescent/affective domain 

of the middle school student? 

c. Did your program prepare you to lead your students into higher order thinking? 

4. Of all the course work you did at the college/university level, which have been the most helpful 
courses to you? Describe for me why? Was a specialized middle level teacher preparation program 
available to you at that time? 

5 . What is distinctive about middle level teaching? With reference to the “earmarks” of developmentally 
responsive middle schools, what should teacher preparation programs be focusing on? 

a. What are the important ideas, principles, or understandings that effective 
middle level teachers need to know ? 
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b. What do they need to know about interdisciplinary-thematic organization of 
curriculum? 

C. What do they need to know about teaming? 

d. What do they need to know about flexible block scheduling? 

e. What do they need to know about advisory programs? 

f. What do they need to know about exploratory programs? 

g. What do they need to know about transition programs? 

6. Are there important characteristics that middle school teachers need to possess or to develop in order to 
be effective? Is there a priority order to this list? Are there characteristics specifically effective in 
working with middle school students in their young adolescent phase of development? 

7. Do your on-the-job experiences help or hinder you in becoming an effective middle level teacher? 
How? Of the many on-the-job experiences you have had, which do you consider to be the most 
helpful? Please describe one. 

8. Since you have begun teaching, in what types of professional staff development programs have you 
participated? Have any been specifically geared toward the middle level teacher? 

a. Have you attended any programs for the cognitive dimension of teaching young 

adolescents? What were they like? 

b. Have you attended any programs for the affective dimension of middle level 

education? What were they like? 

C. Have you attended any programs for high order thinking skills? 

d. In your opinion, what should professional development programs be focusing on? 

9. What does this school do with regard to teacher supervision. . . evaluation. . . teacher mentoring? How 
does it work here? Does it help or hinder you in becoming more effective as a teacher? Please 
comment. 

10. Do you consider yourself (what makes you) an effective middle school teacher today? 

a. What are the signs that you have been effective? 

b. What are the rewards of teaching at the middle level? 



INTERVIEW QUESTIONS: STUDENTS 



Name: 


School: 




Date: 


Ethnicity: 


Gender: 


Grade: 





1 . Tell me about your middle school. What activities do you have here: . . . sports? . . .clubs? . . .anything 
else? Are any of your teachers the coaches or club moderators? 

2. Tell me what it is like to be a middle school student today. What is it like here? 

3. Tell me about your classmates. How would you describe them? How would they describe you? 

4. What classes do you take here? What is one of your favorite classes? Why? What is one of your least 
favorite classes? Why? Do any of your classes or teachers help you explore what you want to learn? 
How? 

5. How many teachers do you have in one day? Do you have a team of teachers? What adjectives would 
you use to describe your teachers? Do they have any similar characteristics? How would your teacher/s 
describe you? 

6. Are middle school teachers different from elementary school teachers? How? 

7. Are any of your teachers more effective (or, “better”) than the others? How do you know? What makes 
that teacher effective/better/good? Can you give me some examples? What makes a teacher “not so 
good”? Can you give me some examples? 
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8. Do you have an advisor? ...an advisory group? Would you go to a teacher if you had a problem? Why 
would you go to a teacher? Why wouldn’t you go to a teacher? 

9. Do the teachers here help you get used to/adjusted to middle school? How? Are they helping you 
prepare for high school? How? 

1 0. What advise would you give a person who wanted to be a Middle School teacher? 

1 1 . What advise would you give a Middle School teacher who wanted to improve as a teacher? 

Can you describe for me the perfect middle school teacher? 
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